改一下llama.cpp把bs>=32时GPU通过PCIe读取主机内存里的模型的逻辑去掉，现在内存里的tensor全部由CPU处理，速度就正常多了。无论是用8G显存还是96G显存的pp性能都各自提升到10倍，远超纯CPU运行的性能。prefill现在可以跑出300 token/s的速度，基本达到日常可用水准。

David's random thoughts

想着五一前后挑战llama4 400B发现并没什么挑战性，q4模型用7970X (150GB/s)纯CPU prefll 108 t/s decode 13.8 t/s，用8G显存offload dense层27 t/s，塞满双卡96G显存能30.8 t/s 不过llama.cpp的override tensors的prefill看起来是用纯GPU走PCIe访问内存里的模型，还有优化空间。至少不应该比纯CPU差

改一下llama.cpp把bs>=32时GPU通过PCIe读取主机内存里的模型的逻辑去掉，现在内存里的tensor全部由CPU处理，速度就正常多了。

无论是用8G显存还是96G显存的pp性能都各自提升到10倍，远超纯CPU运行的性能。prefill现在可以跑出300 token/s的速度，基本达到日常可用水准。

www.tg-me.com/us/David& 39;s random thoughts/com.david_random/575

1.6K viewsMay 1 at 18:52

tg-me.com/david_random/575

Create: 2025-05-01
Last Update: 2025-06-01 09:13:16

BY David's random thoughts

Share with your friend now:
tg-me.com/david_random/575

David& 39;s random thoughts Telegram | DID YOU KNOW?

What is Telegram?

Telegram is a cloud-based instant messaging service that has been making rounds as a popular option for those who wish to keep their messages secure. Telegram boasts a collection of different features, but it’s best known for its ability to secure messages and media by encrypting them during transit; this prevents third-parties from snooping on messages easily. Let’s take a look at what Telegram can do and why you might want to use it.